System and method for placement of augmented reality information for users based on their activity

ABSTRACT

Systems and methods described herein are provided for receiving location information for a real-world object in a user&#39;s augmented reality (AR) view, determining a direction of travel of a user, determining an activity zone shape associated with the direction of travel, responsive to a determination that the real-world object is within the activity zone, rendering on an AR display AR information associated with the real-world object at a location outside an area of the AR display used to display the activity zone. Some embodiments continue to display AR information in the same location if the user changes his or her gaze to read the AR information. Some embodiments render on an AR display subtle highlighting if an object is inside an activity zone and prominent highlighting if an object is outside an activity zone.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/466,600, filed Jun. 4, 2019, which is a national stageapplication under 35 U.S.C. 371 of International Application No.PCT/US2017/066499, entitled “System and Method for Placement ofAugmented Reality Information for Users Based on Their Activity,” filedon Dec. 14, 2017, which claims benefit under 35 U.S.C. § 119(e) fromU.S. Provisional Patent Application Ser. No. 62/437,455, entitled“System and Method for Placement of Augmented Reality Information forUsers Based on Their Activity,” filed Dec. 21, 2016, all of which areincorporated herein by reference in their entirety.

BACKGROUND

When a user is walking around with augmented reality (AR) glasses,applications may display annotations and/or animations overlaid on topof a real-world view and block or obscure a user's ability to see boththe real world and the augmentations.

Different techniques have been considered for determining which ARannotations should be displayed to a user. One such technique is todisplay AR information about real-world (or real-life) objects that arein the users field of view, including objects that may be too far awayto be discerned by the user. Another such technique is to display ARinformation about real-world objects on which the user's gaze rests. Thefirst technique may overwhelm the user with AR information, especiallyin environments where there are many real-world objects in the user'sfield of view for which information is available to be displayed to theuser. The second technique may display AR information about fewerreal-world objects than the first system, though the second techniquedisplays AR information for any real-world object upon which the usermay happen to gaze. The AR information for these objects may interferewith the very scene that the user is viewing.

SUMMARY

Systems and methods described herein relate to user interfaces thatdisplay relevant augmented reality (AR) information in a manner that isunobtrusive to the user and does not interfere with the user's mainactivity. For example, AR information may be restaurant or coffee iconsor directions to such locations. For some embodiments, such ARinformation may be displayed on an optical see-through device as overlaygraphics that highlight a real-world object, such as a sign or abuilding, associated with a restaurant or store. For some embodiments,AR information (such as locations and distances to local coffee shops)may be retrieved from an external server and displayed to a user on anAR display, such as an AR optical see-through device.

A user's main activity may correspond to part of the users visual field(e.g., the center region of where the user may be looking and moving).Systems and methods described herein display augmented informationoutside an activity zone so the information does not interfere with theuser's activity. The user may turn his or her gaze without causingchanges in the placement of the augmented information. Thus, the usermay view and interact with augmented information without the informationintruding into the users main activity. A device or system applicationmay display AR information and objects in a manner that does not obscurean active activity zone.

Systems and methods herein provide dynamic determinations of how andwhen AR information about real-world objects may be displayed to a user.Some embodiments determine how such information may be displayed to auser whose direction of gaze and direction of motion may beapproximately, but imperfectly, aligned without interfering with theuser's interaction with reality.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding may be had from the following description,presented by way of example in conjunction with the accompanyingdrawings.

FIG. 1 is an interface diagram showing an AR system.

FIG. 2 is an example AR display with AR information obstructing a usersview.

FIG. 3 is an example AR display with AR information displayed outside anactivity zone.

FIG. 4 is an example AR display with an activity zone surrounding aperson in a users field of view.

FIG. 5 is an example AR display with AR information displayed along thetop and right side of an AR display.

FIGS. 6A and 6B are perspective view schematic diagrams for theplacement of AR information at two moments in time.

FIGS. 7A and 7B are perspective view schematic diagrams for theplacement of AR information at two moments in time when a user changesdirection of motion.

FIG. 8 is a flowchart showing an exemplary embodiment for determining ascreen location for displaying AR information related to a real-worldobject.

FIG. 9 is a message sequence diagram for messages communicated betweenprocesses and devices for displaying AR information for an exemplaryembodiment.

FIG. 10 is a plan view schematic diagram of a two-dimensional coordinatesystem showing a user and an activity zone.

FIG. 11 is a perspective view schematic diagram of a three-dimensionalcoordinate system showing a user and a real-world object.

FIG. 12 is a perspective view schematic diagram of a three-dimensionalcoordinate system showing a user and two real-world objects.

FIG. 13 is a plan view schematic diagram of an AR display showing tworeal-world objects and associated AR information.

FIG. 14 is a plan view schematic diagram of an AR display showing tworeal-world objects and associated AR information, along with an activityzone and an observation zone.

FIG. 15A depicts an example wireless transmit/receive unit (WTRU) thatmay be used within a communications system.

FIG. 15B depicts an exemplary network entity that may be used within acommunication system.

The entities, connections, arrangements, and the like that are depictedin—and described in connection with—the various figures are presented byway of example and not by way of limitation. As such, any and allstatements or other indications as to what a particular figure“depicts,” what a particular element or entity in a particular figure“is” or “has,” and any and all similar statements—that may in isolationand out of context be read as absolute and therefore limiting—may onlyproperly be read as being constructively preceded by a clause such as“In at least one embodiment . . . .” For brevity and clarity ofpresentation, this implied leading clause is not repeated ad nauseum inthe detailed description of the drawings.

DETAILED DESCRIPTION

In exemplary embodiments described herein, augmented reality annotationsare displayed at a position as not to obstruct a user's view of adetermined activity zone (or region). For example, when a user is movingin a particular direction, the augmented reality annotations may bepositioned so as not to obstruct the user's view in that direction ofmotion. For some exemplary embodiments, the user guides the direction ofmotion, e.g., while walking, riding a bicycle, driving a car, or ridinga Segway. For such embodiments, the users direction of gaze maycontinually approximate the user's direction of motion, even though thetwo directions may not be identical. If a user shifts his or her gaze toanother direction, e.g., to read the AR information or to look at ahighlighted object near the periphery, exemplary embodiments of methodsor systems described herein will not cause AR information or associatedgraphics to move out of the way of the users gaze. Such embodiments mayhave greater stability to the user by displaying information out of theway. When a user is a passenger in a train or car, though, a user'sdirection of gaze may be decoupled from the user's direction of motion.

One embodiment of a method disclosed herein tracks a direction of travelof a user of an augmented reality (AR) display, identifies an objecthaving associated AR information, determines whether the direction oftravel is substantially towards the object, selects a location fordisplay of the AR information, wherein selecting the location comprises:selecting a first location at an offset from the object while thedirection of travel is substantially towards the object, and selecting asecond location that at least partially obscures the object while thedirection of travel is not substantially towards the object, anddisplays the AR information at the selected location using the ARdisplay.

For one or more embodiments, a method for displaying (or rendering) ARinformation about a real-world object in a users view (or AR display)may comprise multiple steps: receiving real-world object data,determining a users activity zone, highlighting a real-world objectbased on where the real-world object lies with respect to the activityzone, displaying a real-world object's AR information outside anactivity zone, and displaying an association of the AR information witha real-world object. An AR system may receive data regarding areal-world object, along with the object's coordinates in a system'svisual field and associated augmented information. For one embodiment,determining a users activity zone may comprise determining the user'sdirection of motion as a straight line and determining a zone as apreset view angle centered on the users direction of motion. For anotherembodiment, determining a users activity zone may be based on anonlinear direction of motion, such as a user walking on a curved path.Determining the user's direction of motion as a straight line maycomprise predicting motion based on (as appropriate): the majordimension of movement of the user, the orientation of the user's body(including chest and feet), and the direction in which the user leans.For one embodiment, highlighting a real-world object based on where areal-world object lies with respect to an activity zone may comprisedisplaying subtle highlighting (such as a silhouette or a soft glowoverlay) for the object if the object is displayed within the activityzone and displaying prominent (or strong) highlighting (such ashatching) for the object if the object is displayed outside the activityzone. For one embodiment, a real-world object's AR information may bedisplayed at a point along the outside periphery of the activity zoneclosest to the object. For one embodiment, displaying an association ofAR information with a real-world object may comprise highlighting the ARinformation in a way that matches the highlighting of the real-worldobject and displaying a line linking the AR information with thereal-world object, which may comprise subtle lines for within-zoneobjects and prominent lines for outside-zone objects.

Prominent highlighting may differ from subtle highlighting in one ormore of various different properties. For example, as compared to subtlehighlighting, prominent highlighting may have one or more of a greateropacity, greater line thickness, greater brightness, greater colorsaturation, or other properties indicative of prominence.

FIG. 1 shows a process interface diagram of an AR system 100. A scenesensor 150, such as a camera or LIDAR system, captures image data. Forone embodiment, scene sensors 150 measure information about a scene 162,e.g., as a point cloud, and communicate such scene information to anActivity Zone Determiner 154 and to augmented reality applications.User-specific sensors 152 may provide information about a user, such asorientation of a users chest or the angle of a users shoes. AugmentedReality applications 160 (or apps) or services external to systems andmethods described herein may receive data related to identifiedreal-world objects along with information to display in an augmentedmanner. An AR application 160 may communicate identified objects andaugmented information 172 to an augmented reality and object locator andinformation placer 158. An Activity Zone Determiner 154 may determine anactivity zone 168 for a scene and communicate information used todescribe the activity zone to an augmented reality object locator andinformation placer 158. A user-specific sensor 152 may measureuser-specific data and communicate user activity information 166 to anActivity Zone Determiner 154. An Activity Zone Determiner 154 may usedata from user-specific sensors 152 to determine a user's activity zone168 comprising of the region of the visual field that may be relevant ormost relevant to a user's activity. For one embodiment, the activityzone may be determined from the user's direction of motion by estimatingwhere the user may arrive within a small duration (such as five seconds)with a diameter corresponding to an inner part of the visual field. Forone embodiment, such an inner part of the visual field may be a “nearperipheral” area that is a wedge-shaped area centered on an axis ofmotion with angles of 30° clockwise and counterclockwise from a centerdirection directly in front on a user's eye. For example, if a user iswalking, an Activity Zone Determiner 154 may determine a zone as thepart of the visual field that includes where the user may be within tenor fewer steps. An Augmented Reality Object Locator and InformationPlacer 158 may locate identified objects with respect to an activityzone 168 and determine subtle or prominent highlights 170, asappropriate. An Augmented Reality Object Locator and Information Placer158 may determine locations outside an activity zone 168 for augmentedinformation 170 associated with these objects that minimizes thedistance between the augmented information 170 and the associatedobjects and may determine a display style for a connection 170 betweenan identified object and an identified object's associated augmentedinformation 170. For one embodiment, object highlight data, augmentedinformation, and connection line display data 170 is communicated to anAR display 156 (or display). An AR display 156 may display augmentedinformation to a user. For one embodiment, augmented information,connection line display data, and object highlight data may be used todisplay data to a user.

For some embodiments, an AR device comprises the components illustratedin FIG. 1: scene sensor, Activity Zone Determiner 154, user-specificsensor(s), augmented reality application(s), Augmented Reality ObjectLocator and Information Placer 158, and an AR display 156. For someembodiments, for example, scene sensors 150 may use a camera integratedinto a smart phone. Other scene and user-specific sensors may beintegrated into a headset or AR glasses. For some embodiments, one ormore scene sensors may be mounted to the top of a headset, along withuser-specific sensors. For some embodiments, mounted to or otherwisecoupled with a headset may be a processor for running software thatimplements an Activity Zone Determiner 154, an Augmented Reality ObjectLocator and Information Placer 158, and one or more AR applications 160.User-specific sensors may measure heartrate, walking or running speed,temperature, and these sensors may be integrated into a smart phone orother AR device. For some embodiments, an Activity Zone Determiner 154and an Augmented Reality Object Locator and Information Placer 158 maybe processes running on a processor within an AR device. For someembodiments, an AR display 156 may be a screen on a smart phone. Forsome embodiments, an AR display 156 may be a see-through display device,such as an AR-enabled vehicle windshield or AR optical see-throughglasses or headset. For some embodiments, scene sensor data 162, 164 ismeasured by scene sensor 150 external to an AR device and communicatedto an AR device. For some embodiments, user-specific sensor data ismeasured by devices external to an AR device and communicated to an ARdevice. For some embodiments, augmented reality applications 160 runningon a device external to an AR device may communicate identified objectsand augmented information to an AR device. For some embodiments, an ARdevice comprises an Activity Zone Determiner 154 and an AugmentedReality Object Locator and Information Placer 158, while AR applications160, scene sensor(s) 150, user-specific sensor(s) 152, and an AR display156 are external.

FIG. 2 shows an example display 200 within an AR device not usingsystems and methods described herein. AR information 202, 204, 206, 208,210, 212 is scattered across the display area and obstructs the centerof the user's vision.

FIG. 3 shows an example display 300 within an AR device using systemsand methods described herein. For one embodiment, real-world objects316, 318, 320 are highlighted either subtly or prominently based onwhether an object is inside or outside an activity zone. For oneembodiment shown in FIG. 3, the big circle is the periphery of anactivity zone 314. AR information 302, 304, 306, 308, 310, 312associated with real-world objects 316, 318, 320 are displayed on theperiphery of the activity zone 314.

FIG. 4 shows an example AR display 400 with an activity zone surroundinga person 402 in a users field of view. AR information 404 regarding theperson's gender, height, and weight are displayed to the left of theperson. AR information 406, 408 related to other objects is displayed tothe right of the person.

FIG. 5 shows another example AR display. For this example, a user iswalking around a city center. AR information is displayed along the topand right sides of the AR display. For this example, icons are displayedshowing the locations of the nearest Ladurée bakery, McDonald's fastfood restaurants, Starbuck's coffee, and taxi stand, along withdistances to each of these locations underneath each icon.

For some embodiments, an AR display is an optical see-through device.For some embodiments, an AR display is provided on a windshield of avehicle. For some embodiments, an AR display is embedded in glasses orgoggles. For some embodiments, an AR display renders all objects andgraphics seen on a screen. For some embodiments, an AR display rendersgraphical overlays on top of optically-seen real-world objects.

For FIGS. 2 and 3, some embodiments may display the example white boxes,circles, and lines as graphical overlays on top of the optically-seenreal-world objects represented by the black and white photograph.Similarly, for FIG. 4, some embodiments may display the text and whiteoutline of the activity zone displayed around the person in theforeground as graphical overlays on top of the optically-seen real-worldobjects represented by the black and white photograph. For FIG. 5, oneembodiment 500 may display a bakery icon 502, two fast food restauranticons 504, 506, a coffee shop icon 508, and a taxi stand icon 510, alongwith distances 524, 526, 528, 530 to each of these locations asgraphical overlays on top of optically-seen real-world objects 512, 514,516, 518, 520, 522 represented by black and white drawings of real-worldobjects. For some embodiments, everything seen in FIGS. 2 to 5 may berendered by an AR display.

FIGS. 6A and 6B are perspective schematic diagrams 600, 650 for theplacement of AR information 604, 654 at two moments in time. At time t₁,a user has been continually walking towards a real-world object 602,shown as a cube in FIG. 6A. AR information 604 related to the object isdisplayed to the left of the object 602 and the users direction ofmotion 606. At time t₂, the user has substantially stopped moving 656towards the object 652, as shown in FIG. 6B. AR information 654 relatedto the object 652 is displayed towards the top of the object 652. Forsome embodiments, responsive to a determination that the user hassubstantially stopped moving in the direction of travel, AR informationassociated with a real-world object may be rendered on an AR display ata location on the AR display that at least partially obscures thereal-world object.

FIGS. 7A and 7B are perspective schematic diagrams 700, 750 for theplacement of AR information 704, 754 at two moments in time when a userchanges direction of motion. At time t₁, a user is walking towards areal-world object 702, shown as a cube in FIG. 7A. AR information 704related to the object 702 is displayed to the left of the object and theuser's direction of motion 706. At time t₂, the user turns to the leftand starts walking to the left of the object 752, as shown in FIG. 7B.AR information 754 related to the object 752 is displayed towards thetop of the object 752 and to the right of the users direction of motion756.

FIG. 8 is a flowchart 800 for an exemplary embodiment for determining ascreen location for displaying AR information related to a real-worldobject. For one embodiment, the direction of motion of the user isdetermined 802. A user frame of reference to location <0, 0, 0> within acoordinate system for the users motion may be set 804, where for oneembodiment, positive x-axis values are to the right, positive y-axisvalues are up, and negative z-axis values are in the direction of theuser's motion. For one embodiment, an activity zone is determined 806 asa cone (or conical region) centered on the z-axis with an inner angle of60 degrees centered on the z-axis and the point of the cone centered onthe user. For one embodiment, the coordinates of a real-world object aretransformed 808 to the coordinate system for the frame of reference ofthe activity zone. For one embodiment, the real-world object'stransformed coordinates are set 808 to <x₀, y₀, z₀>. For one embodiment,the activity zone is further determined 810 to be a cone of depth z₀with a radius r_(c) such that the cone's inner angle is 60 degrees. Forone embodiment, a process determines 812 if x₀ ²+y₀ ²≤r_(c) ². If x₀²+y₀ ²≤r_(c) ², the real-world object is inside the activity zone's coneand subtle highlighting is added 814 to the real-world object on the ARdisplay. Otherwise (for the same embodiment), the real-world object isoutside the activity zone's cone and prominent highlighting is added 816to the real-world object on the AR display.

For some embodiments, subtle highlighting comprises displaying anon-clear transparency on top of the real-world object. For someembodiments, subtle highlighting comprises displaying transparent colorswith an intensity level less than a predetermined threshold. For someembodiments, subtle highlighting comprises displaying a transparentcolor associated with a highlighter, such as light pink, light orange,light yellow, light green, light blue, or light purple. For someembodiments, subtle highlighting comprises displaying a thin-linegraphic around the real-world object or displaying a thin lineconnecting the real-world object to augmented information.

For some embodiments, prominent highlighting comprises displaying anopaque color on top of the real-world object. For some embodiments,prominent highlighting comprises displaying opaque colors with anintensity level greater than a predetermined threshold. For someembodiments, prominent highlighting comprises displaying an opaquecolor, such as solid red, solid orange, solid yellow, solid green, solidblue, or solid purple. For some embodiments, prominent highlightingcomprises displaying a thick-line graphic around the real-world objector displaying a thick line connecting the real-world object to augmentedinformation.

For one embodiment, x_(pos) and x_(neg) are calculated 818 as the twosolutions to Eqn. 1.

$\begin{matrix}{x = {\pm \sqrt{\frac{r_{c}^{2}}{1 + \frac{y_{0}^{2}}{x_{0}^{2}}}}}} & {{Eqn}.\mspace{11mu} 1}\end{matrix}$

For one embodiment, a process compares 820 the distance from <x₀, y₀,z₀> to

$\left\langle {x_{pos},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{pos} \right)},z_{0}} \right\rangle$

with the distance from

$\left\langle {x_{0},y_{0},z_{0}} \right\rangle\mspace{14mu}{to}\mspace{14mu}{\left\langle {x_{neg},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{neg} \right)},z_{0}} \right\rangle.}$

If the distance from

$\left\langle {x_{0},y_{0},z_{0}} \right\rangle\mspace{14mu}{to}\mspace{14mu}\left\langle {x_{pos},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{pos} \right)},z_{0}} \right\rangle$

is smaller than the distance from

${\left\langle {x_{0},y_{0},z_{0}} \right\rangle\mspace{14mu}{to}\mspace{14mu}\left\langle {x_{neg},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{neg} \right)},z_{0}} \right\rangle},$

the augmented reality information for the real-world object is displayed822 at

$\left\langle {x_{pos},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{pos} \right)},z_{0}} \right\rangle.$

Otherwise (for the same embodiment), the augmented reality informationfor the real-world object is displayed 824 at

$\left\langle {x_{neg},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{neg} \right)},z_{0}} \right\rangle.$

For one embodiment, a connection graphic is displayed 826 to connect theAR information to the real-world object.

FIG. 9 shows a message sequence diagram 900 for messages communicatedbetween processes and devices for displaying AR information for anexemplary embodiment. A location is listed above each device or processfor an exemplary embodiment. A Scene Sensor 904 transmits 914, 916 scenedata (such as point cloud) to an Activity Zone Determiner 906 and to anAR Application (which may be an information service). A User Sensor 910may also transmit 918 user information to an Activity Zone Determiner906. For one embodiment, an Activity Zone Determiner 906 determines 920an activity zone and transmits 922 a message describing the activityzone to an AR Object Locator and Information Placer 908. An AR ObjectLocator and Information Placer 908 also may receive 924 a message froman AR Application 912 with location data for real-world objects in ascene and augmented information about those real-world objects. An ARObject Locator and Information Placer 908 determines 926 a location ofthe real-world object with respect to the activity zone and for oneembodiment, generates 928 scene data that may be used to displayaugmented information on the periphery of an activity zone. The ARObject Locator and Information Placer 908 may send 930 a message to anAR Display 902 with scene data augmented with AR information.

FIG. 10 shows a plan view schematic 1000 for a user 1008 and an activityzone 1004. For this example, real-world objects 1010, 1012 are shown asdark, bold circles, while the direction of travel 1002 is shown with anarrow pointing directly up towards a normal width circle. For thisexample, a sample activity zone 1004 is shown as three-dimensional ovalcentered on the direction of activity (or travel). For this embodiment,an inner angle 1006 of 60 degrees is shown for a cone associated withthe activity zone 1004. For this embodiment, the cone's apex is centeredupon the user 1008.

For an exemplary embodiment, an activity zone 1004 is determined using ausers chest and foot orientation based on sensor readings located on ausers body. Another embodiment uses camera image data in theinfrastructure surrounding the user. One embodiment uses one or moremotion sensors, such as a GPS, a compass, an accelerometer, or agyroscope, on a user's device to calculate a main direction of travel.For one embodiment, if an accelerometer or gyroscope is used, a GPS orother location device may be used to calculate a main direction oftravel. For one embodiment, a compass may be used to determine a maindirection of travel. For one embodiment, two or more GPS readings may beused to determine a main direction of travel. An example of techniquesthat may be used to determine a user's direction of travel include thosedescribed in Gabor Paller, Motion Recognition with Android Devices,SFONGE LTD. (Oct. 7, 2011),http://www.slideshare.net/paller/motion-recognition-with-android-devices(see slide 33).

For some embodiments, an activity zone 1004 is calculated based on adirection of travel. For some embodiments, an activity zone's cone 1014has an opening angle is 60 degrees and whose apex is at the center of auser's device (alternatively, the center may be at the users eyes or thedevice's camera) and which is centered about the ray corresponding tothe users direction of motion. The height (in math terminology), whichis the depth in the present setting, of the cone 1014 corresponds to howfar the user may see. For some embodiments, there is a cone 1014 foreach height (e.g., one cone for 3 feet and another cone for 3.1 feet).Some real-world objects project inside and some do not.

FIG. 11 shows a three-dimensional coordinate system 1100 with a frame ofreference for a user 1102 with an origin aligned with the user 1102 andthe z-axis along the user's direction of motion. For some embodiments,coordinates for an activity zone 1108 and coordinates received for areal-world object 1106 may be used with geometric equations to determinewhether a real-world object 1106 is inside or outside an activity zone1108. For one embodiment, the labeling of x, y, and z coordinates followthe Android convention, which are described in Freescale Semiconductor'sApplication Note 4317 found athttp://cache.freescale.com/files/sensors/doc/app_note/AN4317.pdf (seepage 2, FIG. 1). For the example Android coordinate convention, thescreen of the device is perpendicular to ground and facing the user. Thex-axis is to the right; the y-axis is toward the sky; and the z-axis isparallel to the ground and toward the user. Points behind the screenhave negative z-axis values. Some embodiments have the origin at thebottom left of the screen. For an exemplary embodiment used herein, theorigin is in the middle of the screen. The device's screen may be smallrelative to the real world in target applications, so the error betweenthese two embodiments may not be significant.

For some embodiments, scene data is received, which may be a pointcloud. The scene data may correspond to what a user sees naturally inthe real-world environment. For an object in a scene, a ray may existthat goes from the coordinate system's origin to the object. For objectswith a nonzero extent, multiple rays may exist that go from the originto the different sides of the object. For some embodiments, thecoordinate system may be transformed from the received scene data (whichmay be a point cloud) to a coordinate system aligned with the userdirection of motion, which may have negative z-axis values in thedirection of motion. The angle between an object's ray and a z-axisaligned with the user's direction of motion may be calculated. If thiscalculated angle is less than half the opening angle of the cone, thereal-world object may be determined to lie inside the activity zone.Otherwise, for this embodiment, the real-world object may be determinedto lie outside the activity zone. Some embodiments may make thesedeterminations based on calculations using a real-world object'scoordinates without calculating the angle between the two rays describedabove.

For one embodiment shown in FIG. 11, an activity zone 1108 has at leastone cone with an apex at the origin and a base centered on the Z axis.For one embodiment, the radius/height 1104 equals the tangent of 30degrees or 0.577. That is,

$\frac{r_{c}}{z_{c}} = {\frac{x_{c}}{z_{c}} = {\frac{y_{c}}{z_{c}} = {{0.5}7{7.}}}}$

One embodiment has a real-world object at coordinates <x₀, y₀, z₀>. Thisembodiment may have a cone that has the same depth as the object(z_(c)=z₀). For this example, the object is within the user's activityzone if x₀ ²+y₀ ²≤r_(c) ². If the object is within the user's activityzone, the AR information corresponding to the object may be displayed asa point along the outside periphery of the activity zone closest to theobject. For one embodiment, augmented information is displayed at apoint on the periphery of the activity zone closest to the real-worldobject. For one embodiment, this point is determined through use of thePythagorean Theorem and geometric calculation. A line linking thereal-world object to the associated augmented information may bedisplayed between the two items.

FIG. 12 is a three-dimensional coordinate system 1200 centered on a user1202 showing a real-world object 1206 and placement of augmentedinformation. For one embodiment, a real-world object 1206 is at <x₀, y₀,z₀> and the center of a cone's base is at <0, 0, z₀>. FIG. 12 shows oneobject 1214 located inside and one object 1206 located outside anexample activity zone 1208. For this example, both real-world objectsand the cone's base are in the same plane, located at z=z₀, and theequations that follow are for the x-axis and y-axis values. The radius1204 of the cone's base is r_(c). For this example, the slope s of theline from the object to the center of the base of the cone is

$s = {\frac{y_{0}}{x_{0}}.}$

This line intersects the circle at the base of the cone at two points1210, 1212 that satisfy the following Pythagorean Theorem andstraight-line equations 2 to 6:

$\begin{matrix}{{x^{2} + y^{2}} = r_{c}^{2}} & {{Eqn}.\mspace{11mu} 2} \\{y = {s*x}} & {{Eqn}.\mspace{11mu} 3} \\{{x^{2} + \left( {sx} \right)^{2}} = r_{c}^{2}} & {{Eqn}.\mspace{11mu} 4} \\{{\left( x^{2} \right)\left( {1 + s^{2}} \right)} = r_{c}^{2}} & {{Eqn}.\mspace{11mu} 5} \\{x = {\pm \sqrt{\frac{r_{c}^{2}}{1 + s^{2}}}}} & {{Eqn}.\mspace{11mu} 6}\end{matrix}$

For this example, the equations may be used to calculate two solutions,which are the positive (x_(pos)) and negative (x_(neg)) values of x thatlie on the circle and the corresponding values of y, which may bepositive or negative depending upon the slope. For this example, theequation solutions are <x_(pos), (s)(x_(pos))> and <x_(neg),(s)(x_(neg))>. Substituting

$\frac{y_{0}}{x_{0}}$

for the slope s gives

$\left\langle {x_{pos},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{pos} \right)}} \right\rangle\mspace{14mu}{and}\mspace{14mu}{\left\langle {x_{neg},{\left( \frac{y_{0}}{x_{0}} \right)\left( x_{neg} \right)}} \right\rangle.}$

The augmented information may be displayed at the <x, y> location thathas a shorter distance to the real-world object. For this example, thecalculations are the same if the object is inside or outside the cone.

FIG. 13 is a plan view schematic 1300 (or bird's eye view) of a user1306 and two real-world objects 1308, 1312. For one embodiment, agraphic is displayed in front of a user 1308 to indicate the peripheryof the user's activity zone 1304. For another embodiment, a graphic isnot displayed in front of a user 1306 to indicate the periphery of theuser's activity zone 1304. For this example, the users gaze 1302 isaligned with the user's activity zone 1304. For exemplary embodiments,augmented information, except for subtle highlights and linkages to ARinformation displayed along the periphery of the activity zone 1304, isnot displayed within the activity zone 1304. AR information 1310, 1312for remote objects may be displayed outside but near the activity zone.As shown in the embodiment of FIG. 13, AR information 1310, 1314 isdisplayed away from a users gaze direction 1302. One object 1308 locatedwithin an activity zone 1304 and within a user's gaze direction 1302 ishighlighted subtly, while another object 1312 located outside anactivity zone 1304 and outside a user's gaze direction 1302 ishighlighted prominently.

FIG. 14 is a schematic plan view 1400 (or bird's eye view) of a user1406 and two real-world objects 1408, 1412. For this embodiment, auser's activity zone 1404 is displayed aligned with a direction ofactivity 1402 in front of the user 1406, but the users gaze 1416 is offcenter and to the right. The user 1406 may be looking at a highlightedobject 1412 (as shown in FIG. 14), AR information 1414 for an object1412, or an object without highlighting. Similar to the embodiment shownin FIG. 13, the activity zone 1404 remains clear of augmentedinformation except for subtle highlights and linkages to AR informationon the periphery of the activity zone. AR information 1410, 1414 for theremote objects 1408, 1412 may be displayed outside but near the activityzone 1404. The observation zone 1418 is the part of the AR display wherethe user 1406 is looking that is outside of the activity zone 1404. Theobservation zone 1418 may be cluttered with AR information while theactivity zone 1404 remains free of clutter. As shown in the embodimentof FIG. 14, the activity zone 1404 remains stable. For this embodiment,if a user's gaze 1416 shifts, AR information 1414 shown within the scenemay not move (or “jump around”). For this embodiment, the activity zone1404 may become peripheral to the user's vision but the activity zone1404 continues to the clear. For one embodiment, if the user 1406 looksat AR information 1414 outside an activity zone 1404, that ARinformation 1414 does not move (or “bounce away”), which would occur ifAR information 1414 always was displayed outside the users direction ofgaze 1416. The embodiment shown in FIG. 14 uses both an activity zone1404 and an observation zone 1418 in determining where to display ARinformation for a seamless AR user experience.

For one embodiment, an activity zone is determined to be a cone-shapedarea with an apex centered on a user's (or viewer's) head with anopening angle of 60 degrees and a z-axis plane aligned with a displayedobject located at a maximum distance from the user in the direction oftravel (or as far as the user is able to see in the direction oftravel). For one embodiment, no visual demarcation is displayed for theactivity zone, unlike FIG. 3, which shows the activity zone as a thickcircle. For some embodiments, a graphic may be rendered on an AR displaythat indicates a periphery of an activity zone. For one embodiment, anactivity zone may be determined to be a distorted (or non-symmetrical)shape (instead of a symmetrical, conical, or circular shape) based on auser's activity. For example, an activity zone may be wider than theheight if a users activity is focused on the lower part of the scene.For one embodiment, an activity zone is a rectangle. For one embodiment,an activity zone is a plurality of disconnected shapes. For oneembodiment, an activity zone is a three-dimensional shape. For oneembodiment, determination of a user's activity zone is based not on atwo-dimensional model but a three-dimensional model incorporating depth,where the activity zone may be a conical shape or three-dimensional partof the scene. For one embodiment, determination of a user's activityzone may be based, not on the users direction of motion, but on thesalience of a part of the scene, such as, what objects appear in whatpart of the scene. For one embodiment, a users activity zone may haveone or more salient objects based on the users activity. For oneembodiment, an activity zone is not a contiguous region (or is aplurality of disconnected shapes). For one embodiment with real-worldobjects outside of an activity zone, associated AR information isdisplayed (or placed) near the object along with a marker at theperiphery of the activity zone that indicates some AR information isavailable if the user looks toward the real-world object. For someembodiments, additional AR information may be displayed if the userlooks toward current AR information.

Consumer Use Cases

For an example consumer use case, John uses some AR applications as hewalks through New York City. His activity is recognized as walking andan associated activity zone is determined to be the path in front of himup to about 3 feet ahead. John is able to view his walking path becauseAR information received from his AR applications is not displayed in hisdirection of travel. For example, if John sees interesting people, suchas celebrities, some AR apps display information about those people.This information is displayed outside the activity zone.

For another consumer user case, John runs into Mary while walkingthrough Times Square. They have an animated conversation. His activityis determined to be chatting with Mary, and the associated activity zonechanges to Mary's body frame (which may be a silhouette slightly largerthan Mary). As John stands and chats with Mary, he looks left and rightfrom time to time. Different real-world objects may come within vieweither within his activity zone (e.g., next to Mary's ear) or outsidehis activity zone (e.g., to the right and down the street). The ARinformation for these objects is placed outside John's activity zone. IfJohn turns to read this information or to examine a real-world object insome detail, the AR information may change but none of the ARinformation obstructs his view of Mary.

Network Architecture

A wireless transmit/receive unit (WTRU) may be used as a AR displaydevice in embodiments described herein. The AR device shown in FIGS. 15Ato 15B may be used to retrieve AR information.

FIG. 15A is a system diagram of an example WTRU 102. As shown in FIG.15A, the WTRU 102 may include a processor 118, a transceiver 120, atransmit/receive element 122, a speaker/microphone 124, a keypad 126, adisplay/touchpad 128, a non-removable memory 130, a removable memory132, a power source 134, a global positioning system (GPS) chipset 136,and other peripherals 138. The transceiver 120 may be implemented as acomponent of decoder logic 119. For example, the transceiver 120 anddecoder logic 119 may be implemented on a single LTE or LTE-A chip. Thedecoder logic may include a processor operative to perform instructionsstored in a non-transitory computer-readable medium. As an alternative,or in addition, the decoder logic may be implemented using custom and/orprogrammable digital logic circuitry.

It will be appreciated that the WTRU 102 may include any sub-combinationof the foregoing elements while remaining consistent with an embodiment.Also, embodiments contemplate that base stations and/or the nodes thatbase stations may represent, such as but not limited to transceiverstation (BTS), a Node-B, a site controller, an access point (AP), a homenode-B, an evolved home node-B (eNodeB), a home evolved node-B (HeNB), ahome evolved node-B gateway, and proxy nodes, among others, may includesome or all of the elements depicted in FIG. 15A and described herein.

The processor 118 may be a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Array (FPGAs)circuits, any other type of integrated circuit (IC), a state machine,and the like. The processor 118 may perform signal coding, dataprocessing, power control, input/output processing, and/or any otherfunctionality that enables the WTRU 102 to operate in a wirelessenvironment. The processor 118 may be coupled to the transceiver 120,which may be coupled to the transmit/receive element 122. While FIG. 15Adepicts the processor 118 and the transceiver 120 as separatecomponents, it will be appreciated that the processor 118 and thetransceiver 120 may be integrated together in an electronic package orchip.

The transmit/receive element 122 may be configured to transmit signalsto, or receive signals from, a base station over the air interface 116.For example, in one embodiment, the transmit/receive element 122 may bean antenna configured to transmit and/or receive RF signals. In anotherembodiment, the transmit/receive element 122 may be an emitter/detectorconfigured to transmit and/or receive IR, UV, or visible light signals,as examples. In yet another embodiment, the transmit/receive element 122may be configured to transmit and receive both RF and light signals. Itwill be appreciated that the transmit/receive element 122 may beconfigured to transmit and/or receive any combination of wirelesssignals.

In addition, although the transmit/receive element 122 is depicted inFIG. 15A as a single element, the WTRU 102 may include any number oftransmit/receive elements 122. More specifically, the WTRU 102 mayemploy MIMO technology. Thus, in one embodiment, the WTRU 102 mayinclude two or more transmit/receive elements 122 (e.g., multipleantennas) for transmitting and receiving wireless signals over the airinterface 116.

The transceiver 120 may be configured to modulate the signals that areto be transmitted by the transmit/receive element 122 and to demodulatethe signals that are received by the transmit/receive element 122. Asnoted above, the WTRU 102 may have multi-mode capabilities. Thus, thetransceiver 120 may include multiple transceivers for enabling the WTRU102 to communicate via multiple RATs, such as UTRA and IEEE 802.11, asexamples.

The processor 118 of the WTRU 102 may be coupled to, and may receiveuser input data from, the speaker/microphone 124, the keypad 126, and/orthe display/touchpad 128 (e.g., a liquid crystal display (LCD) displayunit or organic light-emitting diode (OLED) display unit). The processor118 may also output user data to the speaker/microphone 124, the keypad126, and/or the display/touchpad 128. In addition, the processor 118 mayaccess information from, and store data in, any type of suitable memory,such as the non-removable memory 130 and/or the removable memory 132.The non-removable memory 130 may include random-access memory (RAM),read-only memory (ROM), a hard disk, or any other type of memory storagedevice. The removable memory 132 may include a subscriber identitymodule (SIM) card, a memory stick, a secure digital (SD) memory card,and the like. In other embodiments, the processor 118 may accessinformation from, and store data in, memory that is not physicallylocated on the WTRU 102, such as on a server or a home computer (notshown).

The processor 118 may receive power from the power source 134, and maybe configured to distribute and/or control the power to the othercomponents in the WTRU 102. The power source 134 may be any suitabledevice for powering the WTRU 102. As examples, the power source 134 mayinclude one or more dry cell batteries (e.g., nickel-cadmium (NiCd),nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion),and the like), solar cells, fuel cells, and the like.

The processor 118 may also be coupled to the GPS chipset 136, which maybe configured to provide location information (e.g., longitude andlatitude) regarding the current location of the WTRU 102. In additionto, or in lieu of, the information from the GPS chipset 136, the WTRU102 may receive location information over the air interface 116 from abase station and/or determine its location based on the timing of thesignals being received from two or more nearby base stations. It will beappreciated that the WTRU 102 may acquire location information by way ofany suitable location-determination method while remaining consistentwith an embodiment.

The processor 118 may further be coupled to other peripherals 138, whichmay include one or more software and/or hardware modules that provideadditional features, functionality and/or wired or wirelessconnectivity. For example, the peripherals 138 may include anaccelerometer, an e-compass, a satellite transceiver, a digital camera(for photographs or video), a universal serial bus (USB) port, avibration device, a television transceiver, a hands free headset, aBluetooth® module, a frequency modulated (FM) radio unit, a digitalmusic player, a media player, a video game player module, an Internetbrowser, and the like.

FIG. 15B depicts an example network entity 190 that may be used withinthe communication system 100 of FIG. 15A. As depicted in FIG. 15B,network entity 190 includes a communication interface 192, a processor194, and non-transitory data storage 196, all of which arecommunicatively linked by a bus, network, or other communication path198.

Communication interface 192 may include one or more wired communicationinterfaces and/or one or more wireless-communication interfaces. Withrespect to wired communication, communication interface 192 may includeone or more interfaces such as Ethernet interfaces, as an example. Withrespect to wireless communication, communication interface 192 mayinclude components such as one or more antennae, one or moretransceivers/chipsets designed and configured for one or more types ofwireless (e.g., LTE) communication, and/or any other components deemedsuitable by those of skill in the relevant art. And further with respectto wireless communication, communication interface 192 may be equippedat a scale and with a configuration appropriate for acting on thenetwork side—as opposed to the client side—of wireless communications(e.g., LTE communications, Wi-Fi communications, and the like). Thus,communication interface 192 may include the appropriate equipment andcircuitry (which may include multiple transceivers) for serving multiplemobile stations, UEs, or other access terminals in a coverage area.

Processor 194 may include one or more processors of any type deemedsuitable by those of skill in the relevant art, some examples includinga general-purpose microprocessor and a dedicated DSP.

Data storage 196 may take the form of any non-transitorycomputer-readable medium or combination of such media, some examplesincluding flash memory, read-only memory (ROM), and random-access memory(RAM) to name but a few, as any one or more types of non-transitory datastorage deemed suitable by those of skill in the relevant art may beused. As depicted in FIG. 15B, data storage 196 contains programinstructions 197 executable by processor 194 for carrying out variouscombinations of the various network-entity functions described herein.

In some embodiments, the network-entity functions described herein arecarried out by a network entity having a structure similar to that ofnetwork entity 190 of FIG. 15B. In some embodiments, one or more of suchfunctions are carried out by a set of multiple network entities incombination, where each network entity has a structure similar to thatof network entity 190 of FIG. 15B.

Note that various hardware elements of one or more of the describedembodiments are referred to as “modules” that carry out (i.e., perform,execute, and the like) various functions that are described herein inconnection with the respective modules. As used herein, a moduleincludes hardware (e.g., one or more processors, one or moremicroprocessors, one or more microcontrollers, one or more microchips,one or more application-specific integrated circuits (ASICs), one ormore field programmable gate arrays (FPGAs), one or more memory devices)deemed suitable by those of skill in the relevant art for a givenimplementation. Each described module may also include instructionsexecutable for carrying out the one or more functions described as beingcarried out by the respective module, and those instructions may takethe form of or include hardware (or hardwired) instructions, firmwareinstructions, software instructions, and/or the like, and may be storedin any suitable non-transitory computer-readable medium or media, suchas commonly referred to as RAM or ROM.

Although features and elements are described above in particularcombinations, one of ordinary skill in the art will appreciate that eachfeature or element may be used alone or in any combination with theother features and elements. In addition, the methods described hereinmay be implemented in a computer program, software, or firmwareincorporated in a computer-readable medium for execution by a computeror processor. Examples of computer-readable storage media include, butare not limited to, a read only memory (ROM), a random access memory(RAM), a register, cache memory, semiconductor memory devices, magneticmedia such as internal hard disks and removable disks, magneto-opticalmedia, and optical media such as CD-ROM disks, and digital versatiledisks (DVDs). A processor in association with software may be used toimplement a radio frequency transceiver for use in a WTRU, UE, terminal,base station, RNC, or any host computer.

What is claimed:
 1. A method comprising: using at least one sensor of anaugmented reality (AR) capable device, determining a current activity ofa user of the AR capable device; determining a direction associated withthe determined current activity; determining an activity zone, theactivity zone being defined by a geometric shape substantially centeredon the determined direction; selecting a position for an AR annotationof a real-world object in the activity zone, wherein the selecting ofthe position is performed such that the AR annotation does not obscurethe activity zone; and displaying the AR annotation on the AR capabledevice at the selected position.
 2. The method of claim 1, wherein thegeometric shape is a 2D shape.
 3. The method of claim 1, wherein thegeometric shape is a 3D shape.
 4. The method of claim 1, wherein thecurrent activity is motion of the user, and wherein the determineddirection is a direction of motion of the user.
 5. The method of claim4, wherein the sensor includes at least one of the following: anaccelerometer, a compass, a GPS, a gyroscope, a camera, or a LIDARsystem.
 6. The method of claim 1, wherein the current activity isassociated with a salient object, and wherein the activity zone isdetermined such that it includes the salient object.
 7. The method ofclaim 1, wherein the current activity is associated with a plurality ofsalient objects, and wherein the activity zone is determined such thatit includes the plurality of salient objects.
 8. The method of claim 1,further comprising: using the least one sensor, determining that thecurrent activity of the user has changed; updating the activity zoneaccording to the changed current activity; and updating the position ofthe AR annotation such that the AR annotation does not obscure theactivity zone.
 9. The method of claim 1, wherein selecting a positionfor an AR annotation for the real-world object in the activity zonecomprises placing the AR annotation a location along a periphery of theactivity zone closest to the location of the real-world object.
 10. Themethod of claim 1, further comprising displaying a graphic connectionbetween the AR annotation and the real-world object.
 11. The method ofclaim 1, wherein the activity zone is a circle, an oval, or a rectangle.12. The method of claim 1, wherein the activity zone is conical.
 13. Themethod of claim 1, further comprising displaying a graphical indicationof the extent of the activity zone.
 14. A system comprising a processorand a non-transitory computer-readable medium storing instructions thatare operative, when executed on the processor, to perform functionscomprising: using at least one sensor of an augmented reality (AR)capable device, determining a current activity of a user of the ARcapable device; determining a direction associated with the determinedcurrent activity; determining an activity zone, the activity zone beingdefined by a geometric shape substantially centered on the determineddirection; selecting a position for an AR annotation of a real-worldobject in the activity zone, wherein the selecting of the position isperformed such that the AR annotation does not obscure the activityzone; and displaying the AR annotation on the AR capable device at theselected position.
 15. The system of claim 14, wherein the currentactivity is motion of the user, and wherein the determined direction isa direction of motion of the user.
 16. The system of claim 14, whereinthe current activity is associated with a salient object, and whereinthe activity zone is determined such that it includes the salientobject.
 17. The system of claim 14, further configured to perform: usingthe least one sensor, determining that the current activity of the userhas changed; updating the activity zone according to the changed currentactivity; and updating the position of the AR annotation such that theAR annotation does not obscure the activity zone.
 18. The system ofclaim 14, wherein selecting a position for an AR annotation for thereal-world object in the activity zone comprises placing the ARannotation a location along a periphery of the activity zone closest tothe location of the real-world object.
 19. The system of claim 14,wherein the activity zone is a circle, an oval, or a rectangle.
 20. Thesystem of claim 14, wherein the activity zone is conical.